PERIOD FUNCTIONS FOR HECKE TRIANGLE GROUPS, AND 
THE SELBERG ZETA FUNCTION AS A FREDHOLM 
DETERMINANT 



M. MOLLER AND A.D. POHL 



Abstract. We characterize Maass cusp forms for any cofinite Hecke triangle 
group as 1-eigenfunctions of appropriate regularity of a transfer operator fam- 
ily. This transfer operator family is associated to a certain symbolic dynamics 
for the geodesic flow on the orbifold arising as the orbit space of the action 
of the Hecke triangle group on the hyperbolic plane. Moreover we show that 
the Selberg zeta function is the Fredholm determinant of the transfer operator 
family associated to an acceleration of this symbolic dynamics. 



1. Introduction and Statement of Results 

Let r be a cofinite Heclce triangle group and consider its action on two-dimensional 
real hyperbolic space H by Moebius transformations. It is well-known [IIcj83, Sel56, 
Fis87] that the zeros with Re s = ^ of the Selberg zeta function 



7 fc=0 



correspond to the eigenvalues of the Maass cusp forms for F. Here the outer product 
is over all closed geodesies 7 on the orbifold r\IHI and ^(7) denotes the length of 
7. In case of the modular group PSL(2, Z) an even closer relation is known. Series 
[Ser85] provided a symbolic dynamics for the geodesic flow on PSL(2,Z)\]HI which 
relates this flow to the Gauss map 



K: 



[0,1]\Q ^ [0,1]\Q 

X ^ |i}=i_|i| 

in a way that closed geodesies on PSL(2,Z)\H correspond to finite orbits of K. 
Mayer [May 76, May90, May91] investigated the transfer operator with parameter 
s G C associated to K. This is the evolution operator of densities on the unit 
interval [0, 1] which are absolutely continuous w.r.t. Lebesgue-s-density under the 
Gauss map K , hence 

{CK,sf){x)^ f 6{x^Kiy))f{y)\dy\-^^Y.('^ + n)-'^f(^). 
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Here S denotes the Kronecker delta function. Mayer showed the existence of a 
Banach space B such that for Res > 5, the transfer operator Ck.s acts on B and 
as such is nuclear of order 0. Moreover, he proved that the transfer operator family 
has a meromorphic extension Ck.s to the whole s-plane with values in nuclear 
operators of order on B. Its possible poles are located at s = (1 — fc)/2, k G Nq. 
Finally, he showed that the Selberg zeta function is represented by the product of 
the Fredholm determinants of ±Ck.s- 

Z{s) ^ det(l - Ck,s) ■ det(l + Ck.s). 

In turn, the zeros of Z are determined by the ±l-eigenfunctions of Ck,s hi B. Efrat 
[Efr93] proved that the values for s such that there exist a ±l-eigenfunction of 
Ck,s correspond to the eigenvalues of even resp. odd Maass cusp forms. Extending 
this result to the level of the eigenfunctions, Chang and Mayer [CM99] as well as 
Lewis and Zagier [LZOl] showed that even resp. odd Maass cusp forms for PSL(2, Z) 
are in linear isomorphism to the ±l-eigenspaces of Mayer's transfer operator (then 
automatically Res = ^). Their proof does not make use of the Selberg trace 
formula. It proceeds in two steps. In a first step they show that Maass cusp forms 
are isomorphic to the period functions of PSL(2, Z) (solutions of a certain 3-term 
functional equation of appropriate regularity; see [Bru97] for an alternative proof, 
and [L('\\'97] for even Maass cusp forms). In a second step they prove that period 
functions and the ±l-eigenfunctions of Ck.s are in linear bijection. 

Each eigendensity / of Ck.s determines a if-invariant measure by /IdAj" (here A 
denotes Lebesgue measure). In turn, certain if -invariant measures on [0, 1] deter- 
mine the zeros of Z and these if-invariant measures with Rc s = i are in bijection 
to the Maass cusp forms for the modular group. More precisely, in this last step 
one has to take limits in s of iiT-invariant measures or densities. 

In this article we set out to establish these relations between Maass cusp forms, 
invariant measures and the Selberg zeta functions for the case that F is any cofi- 
nite Hecke triangle group. With the exception of three groups (namely the Hecke 
triangle groups for q G {3, 4, 6}, cf. Section 2), these Hecke triangle groups are non- 
arithmetic. However, they enjoy a certain symmetry which implies that the Weyl 
law holds for its odd Maass cusp forms [Hej92]. Thus odd Maass cusp forms are 
known to exist. On the other hand, for the non-arithmetic Hecke triangle groups, 
the Phillips-Sarnak conjecture [DIPS85, PS85b, PS85a] states that no even Maass 
cusp forms exist [H<\i92]. 

This work is divided into two parts. In the first part, Section 3, we characterize 
Maass cusp forms for any Hecke triangle group F as solutions of a certain finite-term 
functional equation. This functional equation arises as the characterizing equation 
for the 1-eigenfunctions of a (finite term) transfer operator family Cp.s associated 
to a specific symbolic dynamics for the geodesic flow on T\M. As before, we can 
think of these solutions of the functional equation as period functions for F or, 
alternatively, as densities of i^-invariant measures. When specified to the modular 
group, the arising functional equation is exactly the one used by Lewis and Zagier. 
Thus, we provide here a dynamical interpretation of their functional equation. 

This kind of approach is very sensitive to the symbolic dynamics used. The symbolic 
dynamics we employ here was developed by the second named author in [Fob 10] 
for this specific purpose. It enjoys some special features: The discretization of the 
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geodesic flow it arises from is semi-conjugate to a discrete dynamical system F on 
The map F decomposes into finitely many diffeomorphisms, each of which is 
given by a Moebius transformation induced by an element of F. This implies that 
the transfer operator family Cp.s (and hence the functional equation for the period 
functions) has only finitely many terms. Further, and most crucial, if 51, . . . , (;„ are 
the elements in F used to define F, then the integral transform in [LZOl] applied 
to these elements gives a solution of the functional equation. More precisely, let 
u be a Maass cusp form for F. Then we assign to u and t e R+ a certain closed 
1-form Lu{u,t) (integration against Poisson kernel in Green's form; see Section 3.1 
for details). We define (integration along the imaginary axis) 



Jo 

The special form of the elements gi, . . . , gn allows changing the path of integration 
as follows: 



Transformation properties of these integrals show immediately that -0 satisfies the 
functional equation. Finally, the discrete dynamical system (M''',i^) has a certain 
involutive symmetry. Taking advantage of this symmetry, we can decompose any 
period function into an invariant one and an anti-invariant one. The invariant 
period functions are seen to be in bijection with even Maass cusp forms, the anti- 
invariant ones with odd Maass cusp forms. Now the Phillips-Sarnak conjecture 
translates to the non-existence of non-zero invariant period functions if F is non- 
arithmetic. 

In the second part. Section 4. we represent the Selberg zeta function as the Fredholm 
determinant of a transfer operator family. The transfer operator Cp.s is not nu- 
clear because two of the elements gi,. . . ,gn are parabolic. All the other elements 
are hyperbolic. To overcome this problem, we accelerate the symbolic dynamics 
and hence the discrete dynamical system (K"*", F) on these parabolic elements. Dy- 
namically this is reflected in an acceleration of the discretization, which itself is a 
discretization. For a technical reason we conjugate this acceleration to a discrete 
dynamical system (D,H) on a bounded domain D. The transfer operator family 
Ch,s is seen to be nuclear of order if Res > ^. 

Now another crucial feature of the discretization (M+,F) comes into the picture: 
its coding sequences are unique and all closed geodesies are captured. In other 
words, closed //-orbits are in bijection with closed geodesies. The result is that 
the Selberg zeta function equals the Fredholm determinant of the transfer operator 
family Ch,s- 



The transfer operator family has a meromorphic extension Ch,s to the whole s-plane 
with values in nuclear operators of order and possible poles at s = (1 — A:)/2, k G 
Nq. In turn, the Fredholm determinant of Ch,s provides a meromorphic extension 
of the Selberg zeta function to C with possible poles at s = (1 — fc)/2, k e No, and 
the order of any pole is at most 4. Conjecturally, for Re s = ^, the 1-eigenfunctions 
of the transfer operators Cf,s and Ch,s are in bijection. 





Z{s) = det(l - Ch.s). 
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The discrete dynamical system {D, H) inherits the symmetry of the system (M+, F) 
which is responsible for the splitting of the period functions into odd and even ones. 
Taking advantage of this symmetry, the transfer operator family Ch,s decomposes 
into an even and an odd one, denoted C\j ^ resp. Consequently, the Selberg 

zeta function factors as 

Z(s)=dct(l-/:+ Jdet(l-£^J. 

The symmetry and this factorization extends to the meromorphic extension Ch.s- 
Conjecturally, generalizing the established case for PSL(2,Z), for Res = the 
1-eigenfunctions of ^ should be in bijection with even Maass cusp forms, and 

the 1-eigenfunctions of £^ ^ should correspond to odd Maass cusp forms. Proving 
this conjecture is the last step for the complete transfer operator method for Hecke 
triangle groups. We discuss this in more detail in Section 4.6. In case of the mod- 
ular group PSL(2,Z), the transfer operators Cfj ^ and —Cjj^ are Mayer's transfer 
operator (after the conjugation we used). 

There are two other approaches to represent the Selberg zeta function for Hecke 
triangle groups as a Fredholm determinant of a transfer operator family. The first 
one is provided by Fried [Fn!)(i]. He develops symbolic dynamics for the billiard flow 
on r\S'IHI where F is a triangle group (hence, a subgroup of PGL(2,R)). Then he 
induces it to finite index subgroups via permutation representations, which includes 
a symbolic dynamics for the geodesic flow for Hecke triangle groups. The other one 
is provided by Mayer, Stromberg [MSGS] resp. by Mayer, Miihlenbruch and Strom- 
berg [MMSIO]. For the modular group PSL(2,Z), Bruggeman and Miihlenbruch 
[BM09] show that the 1-eigenfunctions of the transfer operator family of this ap- 
proach are in linear isomorphism to the Maass cusp forms. In [MS08, MMSIO], 
the authors took the generating function for Rosen continued fractions as discrete 
dynamical system and developed a (rather involved) discretization of the geodesic 
flow on F\H. This discretization allows to link the Fredholm determinant of the 
arising transfer operator family to the Selberg zeta function. However, the cod- 
ing sequences are not unique, whence the Fredholm determinant is not exactly the 
Selberg zeta function. Indeed, one has a representation of the form 

Z{s) = /i(s) det(l-£^), 

where /i is a function that compensates for multiple coding. Moreover, results of 
this kind are shown by Pollicott [Pol91] for cocompact Fuchsian groups using the 
Series symbolic dynamics, and by [Mor97] for a wide class of cofinite Fuchsian 
groups (however, not containing Hecke triangle groups) using a modified Bowen- 
Series symbolic dynamics. Also multiple codings exist here, and hence the Selberg 
zeta function is not exactly the Fredholm determinant of the associated trans- 
fer operators. Finally, using representations Chang and Mayer [CMOla, CMOll)] 
extended the case of the modular group to its finite index subgroups and hence 
represented for these groups the Selberg zeta function exactly as the Fredholm de- 
terminant of the transfer operator family induced by the representation. For these 
finite index subgroups Deitmar and Hilgert [DH07] provided period functions for 
the Maass cusp forms of these groups also using representations. For the groups 
ro(A^), the combination of [HMMOo] and [FMM07] shows a close relation between 
eigenfunctions of certain transfer operators and Maass cusp forms. 



HECKE TRIANGLE GROUPS 



5 



Acknowledgments: The authors hke to thank Roelof Bruggeman, Tobias Hart- 
nick, Joachim Hilgert, Dieter Mayer, Martin Olbrich and Fredrik Stromberg for 
helpful comments respective for patiently answering questions about their work. 
Moreover, they like to thank Viviane Baladi for drawing their attention to the 
reference [FriOG], and an anonymous referee for a thorough reading and helpful 
comments. The authors were partially supported by the Sonderforschungsbere- 
ich/Transregio 45 "Periods, moduli spaces and arithmetic of algebraic varieties", 
the Max-Planck-Institut fiir Mathematik in Bonn, and the second named author 
by the SNF (200021-127145). 



2. Preliminaries 



2.1. Hecke triangle groups. Let q G N, q > 3. We consider the Hecke triangle 
group Gq with the presentation 

(T, 5 I 52 = id = {TSy) 

or, equivalently 

{U,S \ S'^ = id ^U"). 
Let A := 2 cos-. Then Gq is identified with the subgroup of PSL(2,M) generated 

by 



or alternatively by 



T:=(J ^) and S := 



U:=TS= ['^ ^] and S. 



The group PSL(2, R) is isomorphic to the group of orientation-preserving isometrics 
of two-dimensional real hyperbolic space. As model for this space we use the upper 
half plane 

H {z e C I Imz > 0} 

endowed with the well-known hyperbolic metric. The action of PSL(2,R) on H is 
given by fractional linear transformations, that is, g = (° ^) G PSL(2,R) acts on 
z g H by 

az + b 

= ;■ 

cz + a 

2.2. Maass cusp forms. A Maass cusp form for the Hecke triangle group Gq is 
a twice continuously differentiable function u: M ^ C which satisfies the following 
properties: It is an eigenfunction of the hyperbolic Laplace-Beltrami operator 

A = - 2 [^ + ^ 

which is constant on each G^-orbit, i. e., for all g G Gq and all z G H we have 

u{g.z) = u{z). 

Moreover, u has at most polynomial growth at infinity. Finally, for all y > it 
should hold that 

»A 

u{x + iy)dx = 0. 
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The eigenvalue of a Maass cusp form is positive and factorizes as s(l — s) with 
< Res < 1. 

2.3. Cohomological characterization of Maass cusp forms. Recently, Brugge- 
man, Lewis and Zagier [BLZ] provided a characterization of the spaces of Maass 
cusp forms for arbitrary discrete cofinite subgroups of PSL(2,R) as various first 
parabolic cohomology spaces. From their work we will use that the space of Maass 
cusp forms for Gq with eigenvalue s(l — s) is isomorphic (as a vector space) to 
the first parabolic cohomology space with values in the smooth and semi-analytic 
vectors of the principal series representation associated to the spectral parameter s. 
They showed that this isomorphism is explicitly given by a certain integral trans- 
form. In this section we expound the parts of this isomorphism which are important 
for the work at hand. 

We use the line model for principal series representation. For the spectral parameter 
s £ C, the space V" •°° of smooth and semi- analytic vectors in the line model of the 
principal series representation associated to s consists of complex-valued functions 
if on M which are both 

• smooth with an asymptotic expansion of the form 

oo 

(1) ip{t) r~.\t\-^''J2cnt-" as|t|^C30 

n=0 

for some complex- valued sequence (c„)„gNo (which depends on Lp), 

• and real-analytic on R \ £' for a finite subset E which depends on (p. 

To define the action of Gq on V" •°° we set 

js{g,t):^{(ct + d)-Y {=\ct + d\-^n 

for s € C, 5 = (° ^) G PSL(2,M) and t e K. Moreover, for a function f:V^R 
on some subset y of M and g G PSL(2, M), we define 

(2) Mg-')f{t)--^Ug,t)fig.t) 

whenever this makes sense. Then the requirement (1) is equivalent to impose that 
the map Ts{S)ip, that is 

extends smoothly to M. The action of Gq on V" is given by Ts (with y = M), 
hence 

{rs{g-')m) ((c< + rfnv(f^) 

for g = (^^) e G,, e Vf'°° and t e M. In the case that c 7^ and t = this 
definition is to be understood as a limit. 

In the following we will recall the definition of the first parabolic cohomology space 
Hp^^.{Gq; '°°) from [BLZ]. Parabolic cohomology spaces are certain subspaces of 
group cohomology. For group cohomology we use the standard model. A 1-cocycle 
in group cohomology of Gq is then represented uniquely by a map a : — ^ 
which satisfies 

Ts{g^^)a{gi,g2) ^ a{gig,g2g) and a{go, gi) + a{gi, g2) ^ a{go, g2) 
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for all g, go, gi, g2 G Gq. A l-coboundary in group cohomology is a 1-cocycle a in 
group cohomology for which there exists a map f : Gq V" such that 

Tsig~^)f{gi) = figig) and a(5o,gi) = /(.9o) - f{gi) 

for all g,go,gi G Gg. A parabolic 1-cocycle is a 1-cocycle a in group cohomology 
which, in addition, satisfies that for each parabolic element p e Gq there is a map 
ip e Vf^°° such that 

a{p, 1) = Ts{p^^)ip - V- 
Here, 1 denotes the unit element of Gq. This property allows to normalize rep- 
resentatives of parabolic cocycle classes to vanish on specific elements. Parabolic 
1-coboundaries coincide with the 1-coboundaries from group cohomology. 

It is convenient to use the notation of restricted cocycles. Because of Gg-equivariance, 
each 1-cocycle is determined by its values on Gq x {1}. Hence we may identify the 
1-cocycle a: G^ V"''°° with the map c: Gq ^ V"''°° given by 

Cg — c{g) :== a{g,l) 

for all g e Gq. Then the space of 1-cocycles of group cohomology of Gq becomes 

Z\Gq-Vf'^) ={c:Gq^ Vf'^ I V5,/i G Gg: Cgh = Ts{h-^)cg + cn}, 
and that of parabolic 1-cocycles is 

Zl,AGq;Vfn = {ceZ\Gq-Vfn I Vp e Gg parabolic 3 e V^*'"": 

The spaces of 1-coboundaries of group cohomology and of parabolic 1-coboundaries 
coincide. They are 

Bl^AG,-,Vfn = B\Gq;Vfn = {.9 ^ r,{g-')i^ - ^ | ^ G V^*'""}. 

Finally, the parabolic 1- cohomology space is the quotient space 

^par(Gg; V" '°°) = Z^^^{Gq]V'^ '°° ) /Bp^j. (Gg ; '°°). 

Theorem 2.1 ([BLZ]). The space of Maass cusp forms for Gq with eigenvalue 
s(l — s) is in linear isomorphism with Hp^,^{Gq;V'^ '°°). 

The isomorphism from Theorem 2.1 is given by an integral transform which we now 
recall. We define the function R : M x H K by 

1 



R{t, z) := Im 



For two eigenfunctions u,v oi the Laplace-Beltrami operator A, Green's form shall 
be denoted by 

, , du dv ^ 

[u, v\ '.= — ■ vaz + u ■ —az. 
oz az 

We choose any zq G H. Then the parabolic 1-cocycle class in Hp^j.{Gq;Vf '°°) 
associated to the Maass cusp form u with eigenvalue s(l — s) is represented by the 
cocycle c with 

(3) c,{t):= [u,R{t,-n 
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If p G Gg is parabolic and a; G M U {00} is fixed by p, then 

(4) m ■■= r[u,R{t,-r] 

J zo 

is in and Cp ~ Ts(j>~^)ip — ip- Here, the integration is along a path in H from 

zq to X where only the endpoints are allowed to be in MU {00}. Changing the choice 
of Zq changes c by a parabolic 1-coboundary. 

To end this section, we show that we may take zg = 00 in (3) to define a rep- 
resentative of the 1-cocycle class [c]. So let zq be any element in H and define a 
representative c: Gg — > of [c] via (3). The parabolic element T E Gq fixes 

X '■= 00. Hence 

with ip as in (4). Since d: g t-^ Ts{g^^)ip — ip is & parabolic 1-coboundary, the 
parabolic 1-cocycle c := c — d represents the 1-cocycle class [c]. As in [LZOl, Chap. 
II. 2] it follows that 

T^ig-') f [u,R{t,-Y] = f 

<J a J g ^ .a 

for a, e H U K U {00} and g e Gq. Thus 

pzo /■g^^.oo poo 

cg{t)^ [u,R{t,-Y]- [u,R{t,-Y]- [u.R{t,r] 

Jg ^.Zq J g i.zo ^ 

= / [u.R{t,-n 

J g 1 .00 

We remark that the integration is performed along a path from g^^.oo to 00 in H. 



3. Maass cusp forms and period functions 

The aim of this section is to show the bijection between solutions of a finite-term 
functional equation and parabolic cohomology classes, stated as Theorem 3.3 below. 
Together with the result of [BLZ] this establishes the bijection between certain 
solutions of the functional equation and Maass cusp forms. The main tool is the 
symbolic dynamics of the geodesic flow constructed in [PohlO] which was designed 
so that 1-eigenfunctions of the associated transfer operator satisfy the functional 
equation. In Section 3.3 we show that the splitting of the solutions of the functional 
equation into an odd and an even part is an immediate consequence of a symmetry 
of the symbolic dynamics and hence of the functional equation. 

3.1. Slow discrete dynamical system. The second named author constructed 
in [PoliKJ] a discretization (symbolic dynamics) for the geodesic flow on the orbifold 
Gg\EI which is semi-conjugate to a discrete dynamical system on (0, 00) \ Gg.oo. In 
this section we introduce this discrete dynamical system and the transfer operator 
family associated to it. Using the integral transform (3) with zq = 00, the geometry 
of the symbolic dynamics resp. of the discrete dynamical system yields immediately 
that Maass cusp forms give rise to 1-eigenfunctions of the transfer operator. 
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We set Dst (0, oo) \ Gq. oo. Recall the elements U and S of Gq from Section 2. 
For fc e N we define 

^--(^^) -^l^-sin(%i.) sin(|.) )■ 

For k = 1, . . . ,q ~ 1 we set 

£'st,fc (3^:^0,5^:^00) n Dst- 
Then the discrete dynamical system developed in [PolilO] is 

(5) F:D,t-^D,u i^b.^,. 5fc for fc = 1, . . . , g - 1. 

We shall refer to the system (Dst,F) as the slow discrete dynamical system. Let 
Fct(Dst;C) denote the set of complex- valued functions on Dgt- The transfer oper- 
ator with parameter s G C associated to F is the map 

Cf,s: Fct(i^st;C)^Fct(At;C) 

given by 

q-l 

In the following we will use Cp s as an operator on Fct(R+; C), which is obviously 
well-defined. Suppose that u is a Maass cusp form for Gq with eigenvalue s(l — s). 
In the integral formula (3) let zq 00 and set (for t E M+) 

/>oo 

:=cs: t ^ / [u,Rit,-y]. 
Jo 

Because g^.O ~ gk+1.00 for k = 1, . . . , g — 2, and 51.00 = 00, 5^-1. = 0, it follows 
that 

(6) ^(t)=/ [u,R{t,-r]^Y. [u,R{t,-r] = J2^s{9kMt). 

Hence ip is a 1-eigenfunction of the transfer operator Cp.s- 

3.2. Period functions and parabolic cohomology. We have just seen that 
any Maass cusp form with eigenvalue s(l — s) gives rise to a 1-eigenfunction of the 
transfer operator Cf,s. In this section we will determine which 1-eigenfunctions arise 
in this way. For s S C we let FEs(M+)5°'^ be the space of functions ip E 
such that tp satisfies the functional equation 

q-l 

(7) i> = Y,rs{gk)^. 

k=l 

and has asymptotic expansions of the form 

00 

(8) m - E as < \ 0, 

00 

(9) ^(0 - t-^' J2 An^"" as t / 00 

m=0 
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with 

(10) C*^ = (-l)™+iL>;„ for all m G Nq. 

We call these functions period functions for Gq. 

Remark 3.1. The asymptotic expansions (8)-(10) are equivalent to the requirement 
that 



m - E as i \ 0, 



m=0 

This is equivalent to that the map 



1 1-^ 



for t e R+, 
^-Ts{S)i}{t) forteR- 

extends smoothly to K. 

Remark 3.2. In the case of the modular group, that is for g = 3, the asymptotic 
expansions (8)-(10) are equivalent to the decay conditions (see [LZOl]) 

-(/;(t)=o(i) (ast\0), V(i) = o(l) (a.st/'oo). 

This is due to the fact that the modular group enjoys some arithmetic features 
which are not valid for other Hecke triangle groups. 

In this section we show that the first parabolic cohomology space H^^j.{Gq \ V" '°°) 
and the space of period functions FEs(R+)J^°'^ are isomorphic. Given a para- 
bolic 1-cocycle class [c] e H^^^{Gq]Vg '°°) there is a unique representative c £ 
Zp^j.(Gg; V" '°°) with = (take (3) with zq = co). To [c] we associate the map 
tl) = V'([c]) : M+ ^ C defined by 

V' := cs |r+. 

Conversely, for ip e FEs(M+);i°'= we define a map c = 0(1/-) : Gq Vf-'°° by ct := 
and 

'-0 onM+, 
~Ts{S)i} onM^. 



Cs 



Theorem 3.3. The map 

FE.(M+)^- ^ Hl,,{Gq-Vfn 
ij) ^ [c(V')] 

defines a linear isomorphism between FEs(R+)^"^'^ and Hp^^{Gq] V" '°°). /is inverse 
is given by 

Hl,,{Gq-Vf^°-) ^ FE,(M+)^- 
[c] ^ ^([c]). 

To prove Theorem 3.3 we have to show that the two maps are well-defined. This is 
split into Propositions 3.4 and 3.6 below. 

Proposition 3.4. Let [c] e Hl^^{G q]Vf ^°°) . Then ^'([c]) G FE,,(]R+);J^^ 
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Proof. Let ip '■= tp{[c])- By definition, 

i'it) = cs{t) 



for a (unique) Maass cusp form u with eigenvalue s(l — s). Here the integral is 
performed along the imaginary axis. One easily sees that this parameter integral 
is real-analytic on R+. Since cs = —Ts{S)cs, Remark 3.1 yields the required 
asymptotic expansions for ip. The discussion at the end of Section 3.1 already 
shows that tp satisfies (7). □ 

For Proposition 3.6 below we need the following lemma. The method of its proof 
is adapted from [BLZ]. 

Lemma 3.5. Suppose that i/j: R+ C satisfies (7). Define (p:R \ {0} — > C fty 



ip := 



-T,{S)i' onR-. 
Then ip satisfies the functional equation (7) onR \ {0}. 

Proof. It suffices to show that ip satisfies (7) on RT . Let t G 
that gf^.t < if and only if 

-.9;"\oo < t < -gjP^.O. 

Suppose that t G ( — g^^ .oo, — g^^^.O). Then 

9-1 g-i 
~^T,{gk)'p{t) = ip>{t) ~^Ts{gk)'p>{t) - Ts{gi)'p>{t) 



One easily checks 



fe=l 



k=l 
k^l 



9-1 



k = l 



TsigiS) 



TsigiS) 



TsigiS) 



TsigiS) 



9-1 



-Ts{Sgi ^S)il; -^TsiSgi '^gk)ip + ip 



k=l 
k^l 



it) 



9-1 



k = l 



it) 



9-1 



k = 
k=tl 

9-1 



it) 



-^TsigkiTp + Ip 



k=l 



it) 



which vanishes because -0 satisfies (7). Therefore (7) is satisfied by <y9 on IR 
Proposition 3.6. Let t/j e FEs(R+)^"'=. Then c{ip) is in Z^^^{Gg;Vf . 



□ 
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Proof. Let c := clip). Obviously, cs is real-analytic on K. up to a finite subset. By 
Remark 3.1, cs and Ts{S)cs extend smoothly to M. Thus, cs determines an element 
in '°° . Hence it only remains to show that c is well-defined. For this it suffices 
to prove that Cg2 and cijq vanish. At first we see that 



cs2{t)^T,iS)csit)+cs{t) 



for t > 0, 



T,{S)^p{t) - T,{S)'4j{t) = fort<0. 



Since C52 is smooth on R, it vanishes for t = as well. Lemma 3.5 and regularity 
yield 

q-l 



oil M. Thus we find 



fc=i 

q-l q-l 

cu, =Y,rsiU-'')cs = CS +Y,rs{U-'')cs 

k=Q k=l 
q~l 

= cs+TsiS)Y,rsiSU-'')cs 

k=l 

q-l 

= cs + Ts{S) ^ Ts{gk)cs 

k=l 

= Cs + Ts{S)cs = 0. 



This completes the proof. 



□ 



3.3. Odd and even Maass cusp forms. A Maass cusp form u is called even 
if u{z) — u(— z) for all z £ H. It is called odd if u{z) = —u{—'z) for all z e H. 
The space of Maass cusp forms decomposes as a direct sum into the spaces of even 
Maass cusp forms and odd ones. Let 



1 

1 



e PGL(2, 



Then [LZOl, Equation (2.4)] (which holds verbatim for Hecke triangle groups) shows 
that the even Maass cusp forms with eigenvalue s(l — s) correspond to the period 
functions tp £ FEs(R+)|^°'^ which satisfy ip = Ts{Q)tp, and the odd Maass cusp forms 
with eigenvalue s(l — s) correspond to the period functions i/j G FEs(R+)jJ°'^ with 
i> = ~Ts{Q)i>. Let 



FE,(M+) 



+ \dcc,+ — 



{V'eFE,(M+), 



dec 



resp. 

FE,(M+)d-'- := {i, e FE,(M+)^- | -r,(Q)V^ V} 

denote the set of even resp. odd period functions. In this section we will show that 
the functional equation (7) and the invariance resp. anti-invariance under Ts{Q) can 
be encoded in a single functional equation in each case. 
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For this we let m := [^J. For q even, we consider the functional equations 

q-l q-l 

(11) ■0 = X! '^s{9k)ip + ^Ts{gm)il; + ^Ts{Qg„i)i' + ^ Ts{Qgk)ip, 

q~l q-l 

(12) V = X! Tsi9k)i^ + ■^Ts{gm)il^ ~ ^Ts{Qgm)il-' - X! '^■■>(Q9k)i^- 

k—m+1 /c— m+1 

For (jf odd, we consider the functional equations 

q-l q-l 

(13) ^=Y. ^-^9k)4' + MQ9k)^^ 

k—m k—in 
q-l q-l 

(14) ^ = X! ^"(fffe)'/' - H TsiQgk)^. 

k—m k—m 

Invoking the crucial identity 

Qgk = gq-kQ for all fc e z, 

which follows from QS = SQ and Q[/ = U^^Q, the proof of the following proposi- 
tion is straightforward. 

Proposition 3.7. If q is even, then 

FEs{R+)i'"''+ = {V e C"^(M+;C) | ^ satisfies (8)-(10) and (11)} 

and 

FEsiR+)t'"''^ = {i' G C"^(R+;C) | ?A safe/?es (8)-(10) and (12)}. 
// g is odd, then 

FEs{R+)t'"''^ = {i' e C"(K+;C) | V safe/?es (8)-(10) and (13)} 

and 

FE,(M+)^"'='- = {V' e C"(R+;C) | V sate/ies (8)-(10) and (14)}. 



4. The Selberg zeta function as a Fredholm determinant 



In this section we show that the Selberg zeta function for Gq equals the Fredholm 
determinant of a transfer operator family arising from a discretization for the geo- 
desic flow on Gq\]HI, which is closely related to the slow discrete dynamical system 
{Dst,F) from Section 3.1. Since the elements 

51= (J and 9q-i=l^\ 

in {Dst, F) are parabolic (the elements g2, ■ ■ ■ ,gq-2 are hyperbolic), the associated 
transfer operator Cf,s is not nuclear and hence does not have a Fredholm determi- 
nant. 

To overcome this problem, we accelerate {Dst,F) on parabolic elements. The out- 
come will be a discretization for the geodesic flow of which the associated transfer 
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operator family Ch,s is seen to be nuclear of order on an appropriate Banach 
space. For its Fredholm determinant we find 

Z{s) = dcl{l - Ch.s). 

Throughout we use the following notation: F is the slow system from Section 3.1, 
G its acceleration and H is conjugate to G by a certain Moebius transformation to 
be defined in (15). 



4.1. Fast discrete dynamical system. In the following we construct the accel- 
eration of the slow discrete dynamical system {Dst,F) on its parabolic elements. A 
crucial feature of the accelerated discrete dynamical system is that it arises from a 
cross section for the geodesic flow on G'g\E[. This in turn is inherited from the fact 
that already {Dst, F) arose from a cross section (see below). 

Let Y := Gq\E-. Suppose that fi is & measure on the set of geodesies on Y. A 
subset CS of the unit tangent bundle of Y is called a cross section (w.r.t. /i) for 
the geodesic flow on Y if /i-almost every geodesic on Y intersects CS infinitely 
often in past and future and if each such intersection is discrete in time. The first 
requirement means that for ^-almost every geodesic 7 on K there is a bi-infinite 
sequence (t„)ngz such that 

lim tn — ±00 

n^±oo 

and 7'(t,i) G CS for all n G Z. The latter means that for each intersection time t, 
that is j'{t) e CS, there exists e > such that ^'{{t - e,i + e)) n CS = {j'{t)}. 
Let TT : SM SY denote the canonical quotient map from the unit tangent bundle 
of H to that of Y. A set of representatives for the cross section CS (w.r.t. n) is a 
subset CS' of 5*11 such that tt induces a bijection CS' -> CS. For each v G SM. let 
7„ be the (unit speed) geodesic on H determined by 7^,(0) = v. 

Let Af denote the set of all unit tangent vectors v G SM such that 7t,(oo) G Gg.oo 
or 7„(— 00) G Gq.oo. Further let NIC ('not infinitely often coded') denote the set of 
geodesies on Y with unit tangent vectors in ■k{M), and suppose that is a measure 
on the set of geodesies on Y such that NIC is a /i-nuU set. The cross section from 
which the slow discrete dynamical system {Dst,F) arises is given as follows [PohlO]: 
A set of representatives is 

— I +b— 

" dx dy' 

This is the set of unit tangent vectors in SM which are based on iM+ and point into 
the right half space {2 G C | Rez > 0} such that the determined geodesies do not 
end or start in a cuspidal point. The cross section is then 

CSf 7r(CS'^). 

It is a cross section w.r.t. hence in particular all closed geodesies intersect CSf 
infinitely often. 

Let Rp be the first return map w.r.t. the cross section CSf, that is the map 

jj> . I CSf — ^ CSf 

V 1-^ Iv'ito) 



CS'f ■■= {a—\,y + b—\,y G SM 



y>0, a>0, beR\\Af 
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where v := (7r|csi,)"^(w), fv ■= t^{Jv) and 

to := min{t > | f„'(t) € CSp}- 

In [Poll 10] it is shown that this minimum exists. Further define r: CS_f 

by t{v) '■= (7u(oo), 7i,(— oo)) and let prj^ : — >■ M be the projection on the first 

component. Then Dst is the image of CS f under prj^ or and F is the unique self-map 
of Dst such that the diagram 



CS, 



CSi 



pri 



Ds 



Dst 

commutes and prj^ or is surjective (for details see [PolilO]). To derive from this the 
accelerated system we set 

u{deCSF\ pr,{T{R-\v))),P^i{r{v)) G (.g-_\.0, g-„\.oo) } 
= e CSf I pT,iT{Rp\d))) e (gr'.0,gr'-o«)} 

These are i?i?-images of the elements w in CSp for which Rf{w) and R\{w) project 
to points in I?st.i resp. Dst,g-i- Let Mg '■= ''■^^(A/'g) and define 

CS'a:^GS'p\NG and CSg CSf \ A/q = 7r( CS'g ) . 

One easily checks the following lemma. 

Lemma 4.1. 

(i) Let fi be a measure on the set of geodesies on Y such that NIC is a fi-null set. 
Then CSg is a eross section w. r. t. fi for the geodesic flow on Y . 

(ii) CS'q is a set of representatives for CSg- 

(iii) Each closed geodesic on Y intersects CSg infinitely often. 

Let Rg be the first return map w. r. t. CSg and let r be defined analogous as before. 
As above, there is a unique discrete dynamical system {Dst, G) such that prj^ or is 
surjective and the diagram 

Rg 



CSr 



CS 



p^i 
Dst 



D. 



G 
r 

2 

pri 

St 
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commutes. To give an explicit formula for the map G we set for n G N, 
and 



Then the map G : Dst Dst is given by the diffeomorphisms 

G'lost,^ — 5fe: ^st,fc ^ i'st for fc = 2, ... ,9 - 2, 

and 

Clc.t.l,,. ■= .91' ■ -C'st,l,n "'^ -^st \ ^st,l 
GlDrt.g-l.n .9g-l ^ £'st,g-l,n ^ At \ £'st,g-l- 

for n G N. We call (I?st, G) the fast discrete dynamical system. 
The transfer operator with parameter s associated to G is 

q-2 

TiSN Ti6N fe=2 

acting on a space of functions still be to defined. Our goal is to find a domain of 
definition on which Cg.s becomes a nuclear operator of order and which contains 
real-analytic functions on of certain decay. Here we encounter the problem that 
we have to work with a neighborhood of [0, oo] in the geodesic boundary of H. 
To avoid changes of charts, we conjugate the dynamical system {Dst,G) and its 
associated transfer operators with the transformation 

<-) -^*(! -/)--^^ 

The transformation T has the following geometric interpretation, which immedi- 
ately reveals all its important properties. Let denote the geodesic closure of H, 
and let D = {z S C I \z\ < 1} be the unit disc model of the hyperbolic plane. The 
transformation T is the concatenation of the Cayley transform C : M — >■ D 



1 1 



1 -i 
1 i 



C 

the rotation r : D D 

r : z I— >■ ze~* "2 

by — ^, and the inverse of the Cayley transform. Hence, T is orientation preserving 
and its inverse is 



The transformed discrete dynamical system (Est,H), called fast discrete dynamical 
system as well, is the following: We set 

^^^^ Est r(-Dst), Esuk ■■= T{Dst,k) for fc 1, . . . , q - 1, 

-Bst.i," ■= T{Dst,i,7i), Est,q^i,n T{Dst,q-i,n) for n G N, 

and 

hk'-^T o gk o T"^ for fc = 1, . . . , q - 1. 
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Then H := Est — >■ Est is determined by 

H\E^,.,--=hk for k = 2,...,q-2 

and 

H\E,t,i,r. '■= ^l-Est,,-i,„ — ^9-1 

for n € N. Note that Est is contained in the bounded set [—1, 1]. The fast discrete 
dynamical system {Est,H) arises from the cross section CSh '■= T(CSg) with the 
first return map Rh ■= T o Rq o T^^. The associated transfer operator family is 

9-2 

neN k=2 raGN 

If / is a A-eigenfunction of £g,s: then Ts{T)f is a A-eigenfunction of Ch,s- We will 
investigate an (analytic) extension of Ch.s- For this we define 

E, :=r((A,c^))= 
and E := £;,_i U -B^ U £^1. Then 

9-2 

nGN A;=2 neN 

defined on a suitable subset of Fct(£'; C), which we will define in the next section. 
Representing each function / : £^ — > C as 

the transfer operator Ch,s is represented by the matrix 

^H.s = Ene^rsiM) EZl^sihk) E„eM^.K-i) • 

VEneN^sK') Elllrsihk) J 

4.2. Nuclearity of the transfer operator. In this section we will construct a 
domain of definition for Ch,s on which this transfer operator becomes nuclear of 
order 0. We use the method of Ruelle [Rue76], which is also used by Mayer in 
[jMay9U]. For this we have to find neighborhoods £k of the sets Ek in C which 
are mapped into each other by the fractional linear transformations that appear in 
Ch.s- Then Ch,s is shown to map a certain nuclear space of holomorphic functions 
on these neighborhoods to a Banach space. As such £h,s is nuclear of order 0. 
Using a continuous embedding of the Banach space into the nuclear space, finally 
defines £h,s as a nuclear self-map. 

We set 
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One immediately checks that hkJ = Jhq-k for fc = l,...,(7— 1. 

Proposition 4.2. There exist bounded connected open subsets £i,£r,^q-i ^ 
with the following properties: 



(i) El C £i, Er C 8r, Eq^l C £q-l, 
(ii) J.Sr ~ ^r, J-^l = 

(iii) fork^2,...,q-2: 



(iv) for n G N; 

(v) for n e N; 

(vi) /or a/^ z G £i we /lawe |(/i^"'")'(z)| < 1, 

(vii) /or a// z G we /laue | < 1, 

(viii) for all z G £i we have Rez > —1, 

(ix) for all z G £q-\ we have 1 > Kez, 

(x) for all z G £r we have 1 > Re z > — 1 . 

Proof. We shall provide explicit sets £i, £r, £q-i with the required properties. To 
that end we set 

2A - 1 5A + 1 cA - 1 



2A + 1' ' 5A-1' cA + 1 

5A + 1 , 2A - 1 , cA - 1 



5A-1' 2A + 1' cA + 1' 

where c > 1 will be specified during this proof. For j G {1, r, q — 1} we define £j to 
be the open Euclidean ball in C whose boundary circle passes through Oj and bj. 
Then (i), (ii) and (viii)-(x) are clearly satisfied. Each of the elements hi, ... , hq^i 
is a Moebius transformation and fixes the extended real axis R U {oo}. Therefore 
it suffices to show that 

(iii') for fc = 2, . . . , 5 — 2, we have 

/i^^.ai, /i^^.l G £r, 
h'^^.Or, h'^^.br, h^^.O G £r, 
h]:^.aq-i,h~^.bq^i,h~^.{-l) G £r, 

(iv') for 7T, G N we have 

/ir"-a^,^r"-^r,/ir"-0 e £i, 

hi". aq^i, hi". bq-U hi". i-l) e £i, 

(v') for 7T, G N we have 

hq"i.ai,h~"-^.bi,h~"^.l G 

hq"i.ar,h-:!^^.br,h-l\.0 G £q-i. 
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Let k d 2, ... ,q — 2. Since h/.^ .E C E,., we clearly have 

ft-^^fli, /ijT^a,., h^^.br, h^^.O, h^^.bq_i e £r 
independent of the choice of c > 1. Further 

A-1 A-r 



h7\i,h7^.{-i)e 



A + 1 A + 1 



Since c > 1, we have h'^^ .1, h'/^^ .( — 1) e £r for any choice of c. In the following we 
show that 1 > h^^ .hi^h^^ .aq^i > —1. If we use the short notation S,k sin(-|7r), 
then 

. -1 ^ 1 f'^^k- Cfc-l ~ (,k+l -Cfe-l+Cfe+l \ 

2sm^\ Hk-i - a+i 2a + 6-1 + a+i J ' 

and hence we have 

, -1 ^ 5A (gfc - gfc+i) - gfc-i + ^k 
" " -5A (a + Cfc+i) + a-i + a ■ 

By addition theorems we get 



(sin (|.) + sin (Mi,)) + sin (^.) + sin (|.) 

= f-SAsin (^tt) +sin(^7r)') + C-5Asin(^7r) +sin(^7r) 



-5A 



< 0. 

Therefore, h^^.aq-i > — 1 if and only if 

_5Asin(^7r) +sin(^^) < 0, 
which is satisfied. Similarly, we see that h^^ .aq^i < 1. Then 

hk'-bi = K'-{J-aq-i) = J-{h--k-a,,) e (-1, 1). 
Hence we can choose c > 1 such that 

cA - 1 , _i 1 , cA - 1 



< h^. .aq^i,h^. .bi < 



for fc e {2, . . . , q - 2}. This proves (iii'). 
For n e N we have h^'"{E) C E^. Thus, 

It remains to show that /i^".ag_i G £i. We have 

5nA^ — 5A — 1 
-"^-^^ SnA^ + SA-l <^<'^- 
Further oi < h^" .aq-^ is equivalent to 

< 2A(10nA2 - 7), 

which is obviously true. Hence, (iv') is satisfied. Then (v') follows from (iv') using 
the symmetry J. 

In order to show (vi). let z G £i. Then | (/ij^^)'(z) | < 1 if and only if 

4-2A2 < |- A2 + 2 + A|^ 
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If we use z = X + iy with a;, y G R, it suffices to show that 

4-2A^ < {2 + X-Xxf. 



From 



and 



it follows that 



5A + f 



2 + A 



{2 + X~Xxf > {2 + X-Xbif 



8A - 2^ ^ 



5A - I 



We now claim that 
This is equivalent to 



50A''' - 20A2 - 34A + 8 > 0, 
which is the case since A > I. In turn, \{h^'^) {z)\ < 1. 
Let w e £q-i- Then z := J.w G Si- Further 

f > liK^Yiz)] = \{h-\jyijw)\ = |(/v\)'Hl- 

Thus, (vii) is satisfied. This completes the proof. □ 

From now on we fix a choice of sets £1, S^, £q-i with the properties of Proposi- 
tion 4.2. For i G {1, r,q~ I}, we let 

'H{£j) := {/: £j C holomorphic} 

and 

U{£) := UiSi) X ^[Er) X n{£q-i). 

Endowed with the compact-open topology, each T-L{£j) is a nuclear space [Gro55, I, 
§2, no. 3, Corollary]. Hence the product space 'H{£) is nuclear [Gro55, I, §2, no. 2, 
Theoreme 9]. Moreover we set 

B{£j) := {/ e 'H{£j) I / extends continuously to £j}. 

Endowed with the supremum norm, B[£j) is a Banach space. Further we let 

(17) B{£) B{£i) x B{£r) x 

be the direct product of the Banach spaces B{£i), B{£r) and B{£q_i). We let Ch.s 
act on %{£) by its matrix representation. 

Proposition 4.3. Let s € C with Res > i. Then the transfer operator Ch,s rnaps 
H{£) to B{£). 
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Proof. Let / e n{£), hence / = {fiJrJg^iV with /, e n{£j). Then 6 := Cn.sf 
with 

9-2 

(18) 6i ^J2^Mfr + 5] r,(/i;'_i )/,_!, 

fe=2 riGN 
9-2 

(19) br^Y. ^^^^l)h +Y.^s{hk)fr + E T.(/i^l)/9-l, 

neN fc=2 n6N 

9-2 

(20) = ^ Ts{K)h +Y.T,{hk)fr. 

nGN fc=2 

We have to show that hi G B{£i), br G B{£r) and G B{£q-.i). 

At first let J G {1, r, q — 1} and fc G {2, . . . ,q — 2}. Clearly, Ts{hk)fr is holoniorphic 
on fj. Since /i^T -Sj C 5^ by hypothesis, 

is well-defined and continuous on £j. Thus, the summand X]fc=2 '^a{hk)fk defines a 
map which is holomorphic on £j and extends continuously to £j. 

Now we consider X^neN '''s('*9-i)/9-i ^^"^ -'^^^ J ^ {li^} ^'^^'^ n G N. 

Since h~\.£j C and | (/i^j':j^)'(z)| < 1 for all z G £q-i by hypothesis, we have 

hq"i-£j C h^\.£j C 
In turn, because /g-i is continuous on £q-i, there exists K >Q such that 

sup{|/q_i(/i^"i.z)| I z G n G N} < K. 
Moreover, the map Ts{hq_i)fq-i is continuous on £j. Further 

4 



J.(V-i,^)= ( 



(nAz + 2 + nA)2 



Thus 

^Re SgTrl Im s\ ^R-c Sg7r| Im s| 



^1 - |nA2 + 2 + nA|2Ro« - |7iA(x + 1) + 2|2R<=s 
we have x ~ Re z > — 1 . L 
Xq := min{Rcz | z G 



where a; := Re z. For z G £j we have a; ~ Re z > — 1. Let 



Then 

^Rc Sg'Ti'l Im s| 



< 



(nA(a;o + l) + 2)'^'=^' 



In turn, 

^Rg s ^7t\ Im s\ 

sup |t,(/i;LJ/,_i(z)| < -— ^,2Rcs --^- 

zsfj (n-A(xo + 1) + 2j 

This shows that the series X^nsN '^s{h^^-i)fq-i converges uniformly on £,j if Re s > i. 
The Weierstrass Theorem shows that 'Ylin&i'^s{h'^-i)fq-i is continuous on £j and 
holomorphic on £j . 



Analogously one sees that X^neN ''"sC^")/! ^® continuous on £r and on £g_i, and 
holomorphic on £r and on £q-i. This completes the proof. □ 
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Proposition 4.4. //Res > ^, the map Ch.s ■ 'H-{£) —> B{£) is nuclear of order 0. 

Proof. According to [Gro^-I, II, §2. no. 1, CoroUaire 4] it suffices to show that the 
image of some neighborhood of in H{£) mider Ch,s is bounded in B{£). We pick 
compact balls Ki C £i^ Kr C £j.^ Kq^i C £q-i, each of which contains more than 
one point. Let M > 0. Then 

Vm {/ e H{£) I sup \f,iz)\ < M, j G {l,r,q~l}} 

is a neighborhood of in T-L{£)- Along the lines of the proof of Proposition 4.3 we 
deduce that 

sup ||£ff,s/||oo 

is bounded. Now [Gro'ir), §2, no. 4, CoroUaire 2] shows that Ch.s is nuclear of 
order 0. □ 

Proposition 4.5. Let Res > ^. Then the transfer operator Ch.s is a self-map of 
B{£) and as such nuclear of order 0. 

Proof. The embedding j: B{£) -> H{£), / iH- /, is linear and continuous. By 
[GroST), II, p. 9], Proposition 4.4 implies that Ch,s' B{£) — > B{£) is nuclear of 
order 0. □ 

4.3. The Fredholm determinant of Ch,s- For Res > ^ the Fredholm determi- 
nant of Ch s is defined as 



det(l - Ch.s) := exp 1 Tr£^,, j 



In order to show that it equals the Selberg zeta function (for Res > 1). we take 
advantage of the following argument by Ruelle [Rue94], linking the Selberg zeta 
function (defined via a continuous dynamical system, namely the geodesic flow) 
to dynamical partition functions (defined via an appropriate discrete dynamical 
system). We may apply it in our situation because each closed geodesic on Y 
intersects the cross section GSh and because his arguments holds verbatim if the 
fixed point sets of the appropriate discrete dynamical system are countable (his 
original argument asks for finite fixed point sets). 

Recall that i^dyn denotes the set of periodic geodesies on Y, and that ^(7) is the 
length of 7 e Pdyn- For Res > 1 let 

Csr{s):= n 

7e-Pdyn 

denote the Smale-Ruelle zeta function. Then the Selberg zeta function becomes 



Zi-s) = l[CsR{-s + k)-\ 



Both zeta functions converge for Res > 1 ([Scl56, Fis87]; note that their con- 
vergence is equivalent), and hence this equality holds on the level of converging 
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products. For v G CSh let % denote the geodesic on Y determined by v (ie., 
7;(0) = u). Then 

rniv) :=niin{t >0 | %it) G CS^} 

is the first return time of v, which exists at least if 7^, is closed. Further, for n € N 
let 

Fixi?^ {deCSH I Rh{v)^v} 

be the fixed point set of i?^. For n G N and s G C, the n-th dynamical partition 
function is defined as 

n-l 

Zn{RHTs):~ JJ^ exp sr/f (as formal series). 

Then [Ruc94] we have the formal identity 



Csr[s) = exp Y] -Zn{RH,S 




In the following we will evaluate the traces Tr ^ and the dynamical partition 
functions Z„ in terms of certain elements of Gq . This will allow us to show that 

Z„(i?H,s) = Tr/:^,,-Trri,,,+i, 

which is the crucial step towards the equality of the Fredholm determinant of Ch,s 
and the Selberg zeta function. The proofs rely on key properties of the slow discrete 
dynamical system and the relation between the coding sequences associated to the 
slow and the fast discrete dynamical system, which we are defining now. 

Let tt: SM — > G'q\S'H denote the quotient map of the unit tangent bundles. For 
V G CSf let V := tt^^(v) D CS'p. Let 71, denote the geodesic on H determined by 
"fy(0) = V. The forward F-coding sequence of v is given by the sequence (a„)„gNo 
where a„ := g^""^ if and only if F"(7„(cxd)) G Dst,k with fc G {1, . . . ,q — 1}. The set 
of arising forward F-coding sequences is denoted hy Ap- 

Analogously, foru G CSh we let v :~ 7r^^(u)nCS^. The forward H -coding sequence 
ofv is the sequence (a„)„gNo where 



an-=h^^ ^ i?"(7„(oo)) G Fst.fc and fc G {2,.. .,(7- 2}, 

On 



'fc 

ft.r" ^ H^-hvioo)) G Est,i,m and m G N, 

^,7"i ^ i^"(7^,(c»)) G Fst,g_i,rn and m G N. 

We denote the set of arising forward i/-coding sequences by Ah- 

Periodic forward coding sequences (in which we are mostly interested) correspond 
in the obvious way to coding sequences (cf. [Poll I (I]), for which reason we will omit 
"forward" when refering to periodic forward coding sequences. A periodic coding 
sequence is denoted by 

(ao, . . . ,afc_i) 

where oq, . . . , Ok-i is a (not necessarily minimal) period of the coding sequence 

(an)nGNo • 
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For n g N we define 

Per_f/,„ := {(ao, . . . ,a„_i) e Ah}, and 
-PH,n := {(ao ■ ■ ■ fln-i)"^ I (ao, . . . , a„_i) g A//}. 

Propositions 4.6 and 4.9 below show that these sets and Fixi?^ are in natural 
bij actions. Moreover they characterize the elements in PH,n- 

Proposition 4.6. For each n G N, the map 

r Fixi?^ ^ Per^f,„ 

1 u I— > H -coding sequence ofv 

is a bijection. 

Proof. For each m g N, the map 

r Fixi?^ ^ {(6o,...,6m-i) g Aj.}^ 
w I— > _F-coding sequence of v 

is a bijection by [PolilO, Corollary 4.158]. Suppose that v g Fixi?^ is such that 
w Tiy) g CSff. Let (6o, . . . , &m-i) be the F-coding sequence of v. Then the 
_ff-coding sequence (afe)fcgNo of ^'^ is periodic and it arises from (6o, ■ • ■ , 6m-i) in 
the following way: For fc = 0, . . . , m — 1 set Cfc := T o o ■ Now collapse the 
successive appearances of hi^ resp. h'^^i in cq, . . . , Cm-i to /ij^" resp. h'^^i In being 
the number of successive /i^^ resp. h'^\). This gives 

ao, . . . , ttp-i 

for a (unique) p < m. Then 

(afc)feeNo = (ao, . . . , ap_i). 

Note that m uniquely determines p, and that this construction is invertible. Since 
each periodic 77-coding sequence arises in this way, the bijections above imply the 
claimed bijections between Fixi?^ and Per H,n- O 

Lemma 4.7. The sub-semigroup ofToGqoT^^ generated by {hi, . . . , hq-i} is free. 
Its parabolic elements are | n g N}. All other elements are hyperbolic. 

Proof. The statement is equivalent to that the sub-semigroup Sq of Gq generated 
by . . . ,gqli\ is free, its parabolic elements are {gi"',gq-i | rt g N} and all 

other elements are hyperbolic. Let g = g^^ ■ ■ ■ g~^^ be an element of Sq such that 
not all Ij = 1 and not all Ij = g — 1. An elementary calculation shows that g 
has only positive entries. By the Perron-Frobenius Theorem, g has a real positive 
eigenvalue of algebraic multiplicity 1 . Hence g is hyperbolic. Then g fixes a unique 
geodesic 7 on with 7(00) > 0. By [PohlO, Propositions 4.182, 4.183], the F- 
coding sequence associated to 7(00) is unique and has the (not necessarily minimal) 
period {gf^ , . . . ,gj'^ ). Moreover, g^" • • • is the unique presentation of g by the 
given set of generators of Sq . In turn, the presentation of any hyperbolic element 
in Sq is unique. The only non-hyperbolic elements are of the form 

(J "1^) ^''^ (nA ?) -itl-'^I^' 
of which the presentation is obviously unique. □ 
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Let n e N and suppose that w ~ so ■ ■ ■ Sn-i is a word in (the symbols) 

{hT,h2,...,hg.2,h['L,\meN}. 

We call w reduced if it does not contain a subword of the form Ji^^h"^'^ or liqliliqli. 
We say that w is regular if it is reduced and also ww is reduced. The length of w 
is n. 

An immediate consequence of Lemma 4.7 is the following observation. 
Lemma 4.8. The map 

{reduced words} — > To Gq o 
So •■ -5,1-1 so---s„_i 

is injective. Further, the image of each regular word is a hyperbolic element. 

For convenience, if sg • ■ • Sn-i is a reduced resp. regular word of length n, then we 
also call the group element sq • • • Sn-i a reduced resp. regular word of length n. 

Proposition 4.9. For each n E N, the map 

(ao, . . . ,a„_i) ^ (flo • • •a„_i)"i 
is a bijection. Moreover, Ph.u is the set of regular words of length n. 

Proof. By Lemma 4.8 it suffices to show that 

P&^'H,n = {(ao, . . . , On-i) | {clq ■ ■ ■ a„_i)^^ is a regular word of length n}. 

One easily sees that the set of all periodic F-coding sequences is the set of all 
periodic sequences {bo, . . . , bm-i) with m e N such that bj € {.9^^, • • • , 9q-i} for 
all j G {0, . . . , m — 1} but not all bj = g^^ and not all bj = g^li. This proves the 
claim. □ 

Let g S PSL(2, M) be hyperbolic. Among the two fixed points of g, one is attractive 
(for the iterated action of g) and the other one is repelling. We denote the attractive 
point by z*{g), the repelling by w*{g). Further there exists (a unique) X{g) G R, 
\X{g)\ > 1, such that g is conjugate in PGL(2,IR) to 

'A(ff) 

Lemma 4.10. Let g G PSL(2, M) be a hyperbolic matrix and suppose that g — ( ° ^) 
is a representative of g in SL(2,M). Then 



z*{g)= 



c w* (g) 

where one has to take X{g) > if and only if Trg > 0. The last formulas are 
understood as z*{g) = oo and w*{g) = if c ~ 0. Further 

g'{z*{g))^X{g)-' and g' {w* {g)) = Xig)' . 
Proof. This is a straightforward calculation. □ 
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Proposition 4.11. For Kes > -k and n G N we have 



'(z*(a))- 



Proof. The set of reduced words of length n decomposes into the subsets 

A^^ of reduced words of length n which begin and end with an element of 
{/l™ I TO G N}, 

A^^ of reduced words of length n which begin with an element of {h™ | m G N} 
and end with an element of {/12, • ■ • , hq-2}, 
^(1 q-i) of reduced words of length n which begin with an element of {/i™ | m G N} 
and end with an element of {h'^_i | to G N}, 

and the sets A^^ ,,), ^(^^la), ^"g-i,r) and ^"^^i^^^i) defined in the 

obvious way. Let 



Then 





An ._ An 
^1 ^(1,1 








An ._ An 
— ^(l,r 


U ^",,,r) U 


and 




4" •— 4" 
9-1 — 


-1) ^"r,g-l) U A"y_ 


-1,9-1)- 


/ 




E ^s(a) 


E ^.(«) \ 












E r,s(a) 


E T-s(a) 


E T-^(a) 






as A;; 






E Ts{a) 


E 


E Ts{a) 


V 






'e^?-i\>ir,-i.,~i) / 



'-'H,s — 



By Proposition 4.9, 

F«,„ = Af^^i) U UA-U U 

By [Gro5G, Chapitre II.2, Prop. 2], 

Trr^^.^Tr ^ r,(a). 

There exist open bounded subsets Fi^Fn^-q-i C C which satisfy Proposition 4.2 
and in addition Tj <^ £j for j G {1, r, (jf — 1}. Let 

B{J^) := B{Fi) X B{Fr) X B{Fq-i) 

be the Banach space defined as in (17). Further let 

B{£) B{T) 

ifl, fr, fq~l) ^ 

denote the restriction map. W.l.o.g. we may assume (possible after applying Rie- 
mann mappings) that for j G {l,r, q — 1} there exist < tTj < gj and zj G C such 
that 



^j^B,^{z,)CB,^iz,)C£^ 
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For all fj e B{£j) we have 



For fc e No let 



and 



fc=o I 



h-> {Z-Zjj 

B{£,) ^ C 

IC-2jl=e3 

<Pfe := {vi.k,Vr.k,Vq-i.k) e -B(f)'. 



Then 



r = (g) efc 

is seen to be nuclear of order 0. Moreover, 

where (pk®Ts{a)ek is understood as ^i,k®Ts{a)ei,k for a € \ ^"j^ -^j (and so on) 
Hence 

Tr/^H.s^Tr ^ Ts{a) = ^ ^ v9fc(T,(a)efc) = ^1 TrTs(a). 
Now [Ruc76] shows 

{a'iz*{a))y 



Tr (a) 



1 -a'(z*(a)) 

This completes the proof. □ 
Theorem 4.12. For Res > 1, we have Z{s) = det(l ~ Ch.s)- 

Proof. Let n e N and v E Fixi?^. Suppose that a is the element in PH,n which 
corresponds to v by Propositions 4.6 and 4.9. Let 

m ■■= min {ft, € N | w € Fixi?^}. 

and let b be the element in PH,m which corresponds to v. Suppose that 7 is the 
closed geodesic on Y determined by 7'(0) = v. Then [PolilO, Proposition 4.153] 
yields that h is the hyperbolic matrix which determines 7, that means any other 
hyperbolic matrix which fixes the endpoints 7(±oo) for any representative 7 of 7 
on H is a power of b. It is well-known that then ^(7) = log N{b) where N{b) ~ A(6)^ 
is the norm of b. By Lemma 4.10, 

m^-\ogb'{z*{b)). 



28 



M. MOLLER, AND A.D. POHL 



Further, p := n/m is a positive integer and a = hP. Thus, 

n-l 

Y,rH{R'hv)=pl{l) = -\oga'{z*{a)). 

k=0 

Therefore, 

Z„{Rh,s)= {a'{z*{a))y (formally). 

Thus, by Proposition 4.11 

Z„(i?ff,s) = TrrA,,-TrrA,,+i. 

This shows in addition that Zn{RH,s) converges for Res > i. Now the claim 
follows as in [May91]. □ 

4.4. Meromorphic continuation. From Proposition 4.11 it follows that the Fred- 
holm determinant dct(l — Ch.s) is holomorphic in {Res > ^}. In this section we 
will show that det(l — Ch.s) extends to a meromorphic function on C with possible 
poles at s = (1 — fc)/2, k e No. The method of proof is adapted from [Ma>'90] and 
[Mor97]. 

We start by proving (Proposition 4.13 below) that the map s n- Ch,s extends to a 
meromorphic function on C with values in nuclear operators of order 0. This means 
that we find a discrete set P C C (candidates for poles) and for each s G C \ P 
we find a nuclear operator Ch,s '■ B{8) — > B{S) of order which equals Ch,s for 
Res > i. Moreover, for each / S B{£) and z g £, the function s i— > Ch.sHz) 
is meromorphic with poles in P. and the map (s, z) i— CH,sf{z) is continuous on 
{C \ P) X £. In Theorem 4.14 below we prove that dct(l — Ch.s) is meromorphic. 

Proposition 4.13. The map s i— > Ch,s extends to a meromorphic function on 
C with values in nuclear operators of order 0. The possible poles are located at 
s ~ (1 — fc)/2, k €z Nq. They are all simple. For each pole sq, there is a neighborhood 
U of So such that the meromorphic extension Ch,s is of the form 

Ch.s = -^As + Bs 

s - So 

where the operators As and Bs are holomorphic on U and As is of rank at most 4. 

Proof. Applying the strategy of [May90] to the four maps 

{Res > i} -> {operators B{£q_i) B{£i)} 



'2 ■ 



^ 1 {Res>i} {operators P(£g_i) -> ^(fr)} 



^ ^ ^ {Res>i} {operators P(£i) ^ P(£r)} 



1 {Res>i} {operators i?(£i) ^ P(f,_i)} 



the claim follows immediately. □ 
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Theorem 4.14. The Fredholm determinant (\ct{\—CH,s) extends to a meromorphic 
Junction on C whose possible poles are located at s ~ {1 — fc)/2, fc G No- The order 
of a pole is at most 4. 

Proof. Let Ch,s be the meromorphic extension of Ch.s from Proposition 4.13 and 
recall the restriction map 

r = ^•^k®ek 
feeNo 

from tjie proof of Proposition 4.11. Set P := {(1 - fc)/2 | fc S No}. On C \ P we 
have Ch.s = Ch.s ° r. Hence 

Tr£^,,== ^ ipk{CH^.sek) 

fceNo 

for each n e N. The latter map is holomorphic on C \ P, thus the Fredholm 
determinant dct(l — Ch.s) is so. 

Let sq e P and pick a neighborhood U of sq such that the operator Ch.s is of the 
form 

Ch.s = -^—As + Bs 
s - So 

with As and Bs holomorphic operators and As of rank at most 4. By [Gro56], 

oo 

dct (1 - Ch^s) = J2{-irTrf\CH,s- 

Since the rank of As is at most 4, evaluating the exterior product shows that for 
each ri G No, the map 

{s- So) Tr/\ Ch,s 

is holomorphic on U . Thus, 

(s-so)-^det(l-£H,,) 
is holomorphic on U as well. This completes the proof. □ 

4.5. Factorization of the Fredholm determinant. In this chapter we show 
that the transfer operator Ch,s and its Fredholm determinant split into two parts, 
one of which conjecturally is related to odd Maass cusp forms, the other one to 
even Maass cusp forms. An easy calculation shows that the transfer operator Ch,s 
commutes with 

/ TsiJ)^ 

J r,(J) 
\rs{J) 

Recall the definition of the sets Est.k, fc = 1, . . . , <?— 1, from (16), and let m :~ L^^J • 
Suppose first that q is even. We define 

q-2 

D y EkUh;^\{-i,oi 

k—7n-\-l 

'■= ( TXT ) ^ U u !)• 

k=2 



, A - 1 ■ 
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Representing functions w. r. t. (x E^^i tXe^ Xe^ 1 1 X-Ei ) ^ the transfer operator Ch,! 
is represented by the matrix 



C 



H,s 







nGN 



nGN 



neN 



9-2 


m 

E ^.(/ife) 


E 










m 




A:— rn 


E 


E ^.(/i? 


9-2 


m 


riSN 




E ^.(/^fe) 


E rsih^l 






riGN 


9-2 

E Ts{hk) 
k—rn 


m 

E rs{hk) 

k=2 






A 



If / is a comnion eigenfunction of Ch,s and 

/ 

J = 



Vs{J) 



rsiJ) 



J 



then / is of the form 



h 

±rs{J)f2 
\±r,{J)hJ 

where one has to take the same sign in the last two entries. Then we find 



/ /i \ 

/2 

±r.(J)/2 



We define 



'-H,s 



/ 9-2 m 

' E Ts{hk)f2 ± E Ts{hkJ)f2 ± E Ts{h'lJ)h 

k^m k^2 neN 

q—2 ni 

E + E r,,(/ife)/2± E r.(/ifc^)/2± E rs{KlJ)h 



\ 



k—rn 
9-2 



fc=2 



riSN 



T16N 



k—m 



k=2 



9-2 



rieN 



E Ts{h^,-l)h + E T,(/lfe)/2± E rs{hkJ)f2 
TiSN fe=m /c=2 



9—2 jn 

± E J) E rs{hk) ± E 

neN fc=m A:=2 

9—2 m 

E ^.(^^i) ± E rsiKJ) E rs{hk) ± E ^.(/^fe^) 

k—m k—2 



\neN neN 

Suppose now that q is odd. Then we set 

A - 1 



Er.b := I 



A + 1 

A- 1 
'ATI 



,0 



9-2 

— U -^st.fc, 
k—ra 

fe=2 
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Representing function w.r.t. (x-B<,-i ; X^r a ; XSr X^i) i the transfer operator Ch,: 
is represented by 

' T.rs{hk) Ersihk) Ersihiy 

k=m k=2 nSN 

9-2 J71-1 

nGM /e^rn A:^2 Ji^N 

q—2 ?n— 1 

E ^.K-i) E rsihk) E ^.(/ifc) E 

raeN fe=7Ti fe=2 nGN 

9-2 m-l 

E ^.(/^9-i) E r,{hk) E ^.(^fc) 

TiGN k=m k=2 



c 



Moreover, as above, 



±T.(J)/2 

v±^.(^)/iy 



We define 



g-2 Tjj.-l 

E Ts{hk)h± E Ts{hkJ)h± E T.(/i?J)/l 

k=m k=2 riGN 

9-2 m-l 



E T,(/i" E T,,(/lfe)/2± E Ts{hkJ)f2± E Ts(/l?J)/l 



fc=2 

9-2 m-l 

E ^s(/^9-l)/l+ E rMh± E T.(/lfcJ)/2± E 

neN fc=m fc=2 nGN 

9-2 m.-l 

rtSN fe=m k=2 , 



c 



H.: 



9—2 ?ri— 1 \ 

rieN fc=m A:=2 

9—2 m-l 

E ^«(/i9-i) ± E ^.(/^?^) E rAhk) ± E ^.(^fcJ) 

neN k=m k=2 



Proposition 4.15. For Res > i, f/ie operators ^ are nuclear of order on 
B{Eq-i) X B{£r) and we have 

det(l - Ch.s) = dct(f - Cjj J ■ dct(f - J. 

Moreover, the Fredholm determinants dct(f — ^) and dct(f — Cjj^) extend to 
meromorphic functions on C with possible poles at s ~ (I — k)/2, fe G Ng. 



Proof. Let 



and 



V2 



1 T,iJ) 
Ts{J) -I 



-1 / 



Tlren V and 7?, are self-inverse and 
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Hence 

dct(l - Ch,s) = det(l - VCh^sV) = dct(l - £+ J • dct(l - TZCJi^^TZ) 
= det(l - £+ J • dct(l - Z:^ J. 

The remaining statements now follow from Proposition 4.5 and Theorem 4.14. □ 

4.6. Comparison of the slow and the fast system. Because of the very geomet- 
ric construction (see [PolilO] and the discussion below) of the slow and fast discrete 
dynamical system, we expect that the following comparison statement holds. 

Conjecture 4.16. Let Res = ^- The space of 1-eigenfunctions of the meromorphic 
extension of Hfj g is linear isomorphic to FEs(IR+)^'''^'+ and hence corresponds to 
even Maass cusp forms. The space of 1-eigenfunctions of the meromorphic extension 
of C]j g is linear isomorphic to FE3(IR~)J^°'^^~ and hence corresponds to odd Maass 
cusp forms. 

A couple of arguments support this conjecture: The functional equation in the def- 
inition of the space FEs(R+)jJ'^^ is the defining equation for the 1-eigenfunctions of 
the transfer operator with parameter s arising from a discretization of the geodesic 
fiow on Gg\H. The transfer operator Ch,s is also associated to a discretization 
of this geodesic flow. The latter discretization is strongly related to the flrst dis- 
cretization, it is a specific acceleration of the first discretization. Heuristically, both 
discretizations and both transfer operator families should contain the same geomet- 
ric information. For Re s = i, the Selberg trace formula implies (via Theorem 4.14) 

that Ch,s has a 1-eigenfunction (more precisely, its meromorphic continuation Ch,s 
has a 1-eigenfunction) if and only if FEs(M+)jJ°'^ is non-trivial. 

For the splitting of the space FEs(M+)5'''^ into the subspaces FEs(M+)jJ,'''^'='= we used 
the symmetry Ts{Q). This splitting is the same as building two transfer operators 

from the slow transfer operator Cf,s along the lines of the construction above. 
The 1-eigenfunctions of Cp ^ are then characterized by the functional equation in 
FEg (M"*" )5°'^'^ ■ The same symmetry (after the necessary conjugation with T) is used 
to decompose Ch.s into the operators -C^^- Invoking the idea that the essential 
geometric properties are captured by both discretizations, the 1-eigenfunctions of 

(more precisely, of should correspond to the 1-eigenfunctions of Cp g, 

resp., of appropriate regularity. 

For the case q ~ 3, the transfer operators Cj^ ^ are ± X^neN '^)' which is 
Mayer's transfer operator [May91] (up to the conjugation by T). In this case. 
Conjecture 4.16 has been established by Efrat [Efr93] on the level of parameters s 
for the transfer operators and eigenvalues s(l — s) for Maass cusp forms, and by 
Chang and Mayer [C'M99] as well as by Lewis and Zagier [LZOl] on the level of 
eigenfunctions and Maass cusp forms. 

To end, we prove two lemmas which might be helpful in proving Conjecture 4.16. 

Lemma 4.17. If f ~ (/g_i,/r) is a 1-eigenfunction of C\j ^ or Cj^ ^, then fr is 
determined by fq^i- 
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Proof. W.l.o.g. suppose that q is odd and / is a l-eigenfunction of Ch,s- Then 

q-2 m-1 

fq-1 = J2 rs{KJ)f<l-l + E ^-ihk)fr + MhkJ)fr 
neN k=rn k=2 

q-2 m-l 

fr = Y. ^s{h"g-l)f,-l + E ^s{KJ)h-l + E ^s{hk)fr + E rs{hkJ)fr. 

neN riGN fc=m fc=2 

On £q-i n which is an open non-empty set, we have 

= E ^.(^^i)/<?-i- 

TieNo 

Since fr is holomorphic, it is determined by its values on H £r- D 

The solutions of the functional equations (11)-(14) are determined uniquely by 
their values on a "fundamental domain", as given in the following lemma. As in the 
corresponding discussion in [LZOl] it is not clear how to recognize the functions on 
the fundamental domain that extend to functions on with sufficient regularity 
to lie in FE,(M+)f,<='='±. 

Lemma 4.18. Let tp be a solution to one of the functional equations (11)-(14). 
Then ijj is determined by its values on the interval [1, 1 + A]. 

Proof. We suppose that q is odd and tp satisfies (13) (all the other cases are analo- 
gous). Then tfj = Ts{Q)ip. Hence it suffices to show that ip is determined on [1, oo). 
The functional equation (13) applied to x — A gives 

(21) i;{x) = V(x - A) - js(5r'g, X - A)V^ ( ^— + A 

7n — 1 7n — 1 

- E '^o^Q9k)^{x - A) - ^ Ts{gk)ip{x - A). 

fe=2 k=2 

Now 

rn— 1 

[1,1 + A]D U g^\R+ 

k=2 

and 

1 



+ Ae [i,i + ?iA] 



a; — A 

for .T e [1 + nA, 1 + (?! + 1)A] and n e N. Thus, iteratively applying (21) to the 
intervals [1 + A, 1 + 2A], [1 + 2A, 1 + 3A], . . ., determines ip on [1, oo). □ 
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